The credibility of scientific research is in doubt, among lay consumer (Hornsey & Fielding, 2017) and scientist (Pashler & Wagenmakers, 2021) alike. Several tools have been proposed to combat this “crisis of confidence” (Ibid., p. 528). One such tool is the crowd science approach: “the organization of scientific research in open and collaborative projects” (Franzoni & Sauermann, 2014, p. 1). We focus on crowdsourced data analysis, also known as the many analysts or multi-analyst approach: giving the same dataset to different teams of scientists, who independently analyze it to answer the same research question and/or estimate a parameter of interest.
\(~~~~~\) According to science reformers, crowd-scientific findings that tell a consistent story should garner more confidence in the conclusions and increase public faith in science (Silberzahn et al., 2018; Uhlmann et al., 2019). Here, we ask if we can find empirical evidence for these claims: Does crowdsourcing data analysis improve lay perceptions of scientific findings?
Objectives
We explore the effects of scientific findings emerging from a crowd of researchers (vs. a typical research collaboration) on lay consumers’ posterior beliefs, perceptions of credibility, confidence in an aggregate effect size estimate, and ratings of researcher bias, error, and discretion.
\(~~~~~\) We compare the effects of providing lay consumers with a single, aggregate parameter estimate (the single-analyst condition) vs. multiple parameter estimates that (a) vary slightly and are all positive, leading to the same qualitative conclusion (the multi-consistent condition) or (b) vary widely and are of both signs, leading to differing qualitative conclusions (the multi-inconsistent condition). In all three conditions, the given estimates average to 5%.
Preregistered Hypotheses
Table 1: Predicted direction of effects
| Measure | Multi-consistent | Multi-inconsistent |
|---|---|---|
| 1. Posterior beliefs | ||
| 2. Credibility | ||
| 3. Confidence | ||
| 4. Bias | ||
| 5. Error | ||
| 6. Discretion | No prediction | No prediction |
Note. Table 1 indicates the predicted direction of effects for all outcomes, compared to the single-analyst condition and controlling for prior beliefs (a green plus/red minus indicates a positive/negative prediction, respectively). For example, we hypothesized that, compared to a single-analyst study and controlling for prior beliefs, ratings of credibility would be greater in the multi-consistent condition and lower in the multi-inconsistent condition.
We run an experiment (N = 1,498) with three conditions
\(~\) Single-analyst
A single, aggregate parameter estimate
\(~\) Multi-consistent
Multiple parameter estimates with low variance and high consensus
\(~\) Multi-inconsistent
Multiple parameter estimates with high variance and low consensus
Experimental Design
Figure 1: Estimates (and 95% CIs) for all outcomes
In line with our hypotheses, lay consumers of multi-analyst studies with inconsistent results
\(~~~\) Have lower posterior beliefs
\(~~~\) Find the results less credible
\(~~~\) Have less confidence in the average effect size estimate
\(~~~\) Believe the results are more likely to stem from bias
\(~~~\) Believe the results are more likely to stem from error
Contrary to our hypotheses, lay consumers of multi-analyst studies with consistent results
\(~~~\) Have lower posterior beliefs
\(~~~\) Believe the results are more likely to stem from error
We found no significant effects on
\(~~~\) Credibility of the results
\(~~~\) Confidence in the effect size estimate
\(~~~\) Ratings of bias
Exploratory results
For the additional exploratory measure, lay consumers of multi-analyst studies (both with consistent and inconsistent results)
\(~~~\) Perceive greater researcher degrees of freedom
Figure 2: Distribution of prior and posterior beliefs by condition
Conclusion
\(~\) Crowdsourced data analysis has many worthy uses, but…
\(~~\) Variability and lack of consensus may evoke negative responses
Future Directions
\(~~~\) Perceptions of scientists?
\(~~\) Science communication and communicating uncertainty
\(~~~~\) Other suggestions?
Open Science Statement
The preregistration, survey materials, data, and code that support the findings of this study are openly available on GitHub and the OSF.
\(~~\) [Insert GitHub link here]
\(~~\) [Insert OSF link here]